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Abstract 

In this paper we deal with Radon transforms for generahzed flag manifolds 
in the framework of quasi-equivariant P-modules. We shall follow the method 
employed by Baston-Eastwood and analyze the Radon transform using the 
Bernstein-Gelfand-Gelfand resolution and the Borel-Weil-Bott theorem. We 
shall determine the transform completely on the level of the Grothendieck 
groups. Moreover, we point out a vanishing criterion and give a sufficient 
condition in order that a P-module associated to an equivariant locally free 
C-module is transformed into an object of the same type. The case of maximal 
parabolic subgroups of classical simple groups is studied in detail. 

Introduction 

Let G be a reductive algebraic group over C, P and Q two parabolic subgroups 
containing the same Borel subgroup of G. Let X = G/P, Y = G/Q, and let S be 
the unique closed G-orbit in X x y for the diagonal action. Then we can identify 
S with G/P r\Q. The natural correspondence 

X ^ S ^Y, 

where / and g are the restriction to S of the projections of X x F on X and Y, 
induces an integral transform from X to y which generalizes the classical Radon- 
Penrose transform. This subject has been investigated intensively both in the com- 
plex and real domains (see e.g. Baston-Eastwood [1], D'Agnolo-Schapira [5], Kakehi 
[6], Marastoni [10], Oshima [12], Sekiguchi [14], Tanisaki [15]). 

Our aim is to study this transform in the framework of quasi-G-equivariant V- 
modules (see Kashiwara [7]), i.e. the functor 

R : BliVx) - D^(Py), RiM) = gT'M, (0.1) 
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where Dg,(D.) denotes the derived category of quasi-G-equivariant X>- modules with 
bounded cohomologies, and and /"^ are the operations of direct image (integra- 
tion) and inverse image (pull-back) for P-modules. More precisely, we consider a 
Px-module of type A4 = VC = Vx ®Ox where C is an irreducible G-equivariant 
locally free Ox-module. In this case it is easily seen that 



HP{R{VC))^0 foranyp<0 (0.2) 

(see Lemma 1.4 below). Note that the Grothendieck group of the category of quasi- 
G-equivariant Px-modules of finite length is spanned by elements corresponding to 
the objects of the form T>C 

As in Baston-Eastwood [1] our analysis relies on the Bernstein-Gelfand-Gelfand 
resolution and the Borcl-Wcil-Bott theorem. Using the Bcrnstcin-Gclfand-Gclfand 
resolution in the parabolic setting (see Bernstein-Gelfand-Gelfand [2], Lepowsky [9], 
Rocha-Caridi [13]) we obtain a resolution of the quasi-G-equivariant "Dg-module 
r\VjC) of the form: 

rn ro 

^ 0P£„, ^ . . . ^ ^VLok ^ r\V/:) ^ 0, (0.3) 

k=l k=l 

where Cik are irreducible G-equivariant locally free Og-modules (see § 2.2 for the 
explicit description of Cik). Then we have 

g_^{VCik) = Vy R9*{J~'ik ^3) 

by the definition of g^, where fig denotes the sheaf of relative differential forms with 
maximal degree along the fibers of g. Moreover, the Borel-Weil-Bott theorem tells us 
the structure of Rg^{Cik^^^flg). In particular, we have either Rg^{Cik®Q^flg) = or 
there exist a non-negative integer rriik and an irreducible G-equivariant Oy-module 
C'^^. such that Rg^{Cik (8)0^ ^g) — C'^^l—mik]. Thus setting 

I = {{i, k);Q^i^n,l^k^ri, Rg^idk ^9) ^ 0}> 

we have 

„ (T>r \ - ! '^^iki-^ikl {{i, k) e J), , . 

(see §2.2 below for concrete descriptions of X and Cik-rriik for {i, k) G X). 

Then we can study the structure of R{VC) = gJ^^(VC) using (0.2), (0.3) and 
(0.4). For example we have the following result. 

Theorem 0.1. Let the notation he as above. 
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(i) We have 




in the Grothendieck group of the category of quasi-G-equivariant Vy -modules. 

(ii) //X = 0, then R{VC) = 0. 

(iii) If I consists of a single element {i, k), then R{T>C) = T>C^fJ[i — niik]- 

(iv) If i^ ruik for any {i, k) e I, then we have Hp{R{VC)) — unless p — 0. 

(v) If i > rUik for any (i, k) E X with i > and if thqi — 0, then there exists an 
epimorphism VyJOqi H^{R{T>C)) {note that r^ — l ). 

Assume that C is invertible and that there exists a G-equivariant invertible Oy- 
modulc C satisfying f* C®^^VLg = g*C'. We call such a pair (C, C) an extremal case 
for the correspondence (if P U Q generates the group G and if G is semisimple, then 
there exists a unique extremal case). In this case there exists a natural nontrivial 
"Dy-linear morphism 



Theorem 0.2. Let (£, C) be an extremal case. 

(i) We have Hp{R{VC)) — for any p ^ if and only if i ^ rUik for any 



(ii) Assume that H^{R{VC)) = for any p 0. Then $ is an epimorphism if 
and only if i > rUik for any (i, k) G T with i > 0. 

(iii) Assume that H^^RiVC)) = for any p 0. Then $ is an isomorphism if 
and only if I consists of a single element (0, 1). 

We do not know an example of an extremal case (£, £') such that H^{R{T>C)) ^ 
for some p ^ 0. We have checked that H^{R(T>C)) = for any p 7^ by a case- 
by-case analysis when G is a classical simple group, P, Q are maximal parabolic 
subgroups and (£, £') is the extremal case. In general the morphism $ for an 
extremal case (£, C) is not necessarily an epimorphism nor a monomorphism. It 
would be an interesting problem to determine the kernel and the cokernel of $. 

The transform of a D-module, a problem of analytic nature, is not sufficient to 
cover the general problem of integral geometry. In order to do this, one should couple 
the transforms in the frameworks of P-modules and sheaves. This is better described 
in the adjunction formulas (see D'Agnolo-Schapira [5]), and we shall briefly discuss 
this point with an example in the case of G = SLn+i{C). 

We would like to thank M. Kashiwara for useful conversation on quasi-equivariant 
P-modules. 



^■.VC H\R{VC)). 



(0.5) 



(i, k) e I. 
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1 Preliminaries on 'D-modules 
1.1 Functors for "D-modules 

Let Z be an algebraic manifold (smooth algebraic variety) over C. We denote by 
Oz the structure sheaf, by Qz the invcrtiblc O^-modulc of differential forms of 
maximal degree, and by Vz the sheaf of differential operators. In this paper an 
O^-module means a quasi-coherent C^-module and a "D^-module means a left T>z- 
module which is quasi-coherent over Oz- We denote by Mod(X>z) the category of 
"Dz-niodules and by ^^{Vz) the derived category of Mod{T>z) whose objects have 
bounded cohomology. 

If / : Z — > Z' is a morphism, we set 

— Qz/z' — ^z ®f-iOzi f ^z' 'i 
and, for an C^'-module C, we set 

fc' = Oz ®f-.o„ r'^', Lrc = Oz f-'c. 

We denote by and the direct and inverse image for left "D-modules: 

: B\Vz) - Db(P^O, = Rf,{Vz'^z M), 

/-I : Db(P^,) ^ Db(P2), r^M' = Vz^z' ^f.^^^, f-'M', 

where a (P^, /~^r'2')-bimodule T>z^z' and an (/"^D^^, D^j-bimodule Vz'^z are 
defined by 

T^Z-^Z' = Oz ®f-iOz, f '^Z', 'Dz'^z = ^Z ®Oz '^Z-^Z' ^f-io^, f ^z' ■ 

Note that for a P^'-module M. we have f^^M. ~ Lf*M. as a complex of Oz- 

modules. Note also that we have canonical morphisms Oz — > Vz-^z' and ^1/ — > 
Dz'^z of O^-modules. 

The following result is well-known and easy to prove. 

Lemma 1.1. Let fi : Z ^ Xi and f2 ■ Z ^ X2 be morphisms of algebraic mani- 
folds. 

(i) We have 

Vx.^z Vz-.x, ^/r'J^Xi 0^-10^^ {Vx,><x,^z Oz). 

(ii) Assume that Z ^ Xi x X2 is an embedding. Then we have 

Vx^^z 'Dz-^Xi = 'Dx^^-z ®t:)z '^z-^Xi, 

and the canonical morphism — 'Dx2*-z <H)x>z '^z-^Xi of {f 2^0x2, fi^Oxi)- 
bimodules is a monomorphism. 
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For a locally free Oz-module £, we set 

VC = Vz ^' 

and for a closed submanifold Z of an algebraic manifold X we define a "Dx-module 
Bz\x supported on Z by 

where d — codimx Z and i : Z ^ X denotes the embedding. 
1.2 Radon transforms 

Let X and Y be algebraic manifolds over C, and denote by qi and q2 the projections 
of X X y onto X and Y respectively. Let be a locally closed submanifold oi X xY 
and let i : 5 — s> X X y be the embedding. The geometric correspondence 

xJ-S-^Y (1.1) 
where / and g are the restrictions of qi and ^2, induces a functor 

R : Db(Px) ^ Db(I?y), = gX~\M), (1.2) 

called the Radon transform. 
Lemma 1.2. Lei 6e a Vx-module. 

(i) W^e /lave 

i?(A<) = Rg^iVy^s ^^5^x) f-'M) 

(ii) // S is dosed in X x Y, then we have 

R{M) = ?2,(gi-'M Bs\xxy). 

Proof, (i) follows from the definition and Lemma 1.1, and (ii) is a consequence of 
the projection formula for P-modules. □ 

Let us consider the special case where M. — VC — T>x (E)^^ JC. By Lemma 1.2 
we have the following. 

Lemma 1.3. Let C be a locally free Ox -module. 
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(i) We have 

R{VC) = Rg.iiVy^s ®vs '^s^x) ^f-^o^ f'^C) 

(ii) If S is closed in X x Y , then we have 

R{V£) = Rq24q^\nx >C) (^^-i^^ Bsixxy)- 

An immediate consequence of Lemma 1.3(i) is: 

Lemma 1.4. For any locally free Ox -module L we have H'^{Rf(DO)) — for any 
p<0 . 

Definition 1.5. Let C (resp. C) be a locally free Ox- (resp. Oy-) module. Wc say 
that the pair {C,C') is an extremal case for the correspondence (1.1) if there is an 
O^-linear isomorphism 

Proposition 1.6. Let (C.C) be an extremal case for (1.1). Then there exists a 
natural nontrivial Vy -linear morphism 

VC i/°(i?(P£)). (1.3) 

Proof. The canonical morphism D,g — > Vy^s T^s-^x induces a monomorphism 

g*C' ~ VLg f-^C ^ Vy^s ®vs T^s-.x 0f-iox f~^C 

of g'~^Oy-modules. Applying we obtain a sequence of morphisms 

C ^ C ®^^g,Os^g.{g*C') 

^ g.{Vy^s Vs-.x ^f-iox f-'C) = H%R{VC)) 

of Cy-modules. The morphism C — > jC'<S)Q^g*Os is nontrivial by the definition, and 
the morphism g^{g*£') — > g^lVy^s ®x)s '^s^x f~^^) is a monomorphism by 

the left exactness oi g^,. Thus the composition C H^{R{VC)) is nontrivial. Hence 
it induces a canonical nontrivial morphism VC H^{R{T>C)) of Py-modules. □ 

1.3 Adjunction formulas 

In this subsection we consider topological problems, and hence we work in the ana- 
lytic category rather than the algebraic category. 

For a complex manifold Z we denote by Oz the sheaf of holomorphic functions 
on Z and by Vz the sheaf of holomorphic differential operators. For an algebraic 
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manifold Z over C we denote the corresponding complex manifold by Zan, and 
for a morphism / : Z — > Z' of algebraic manifolds we denote the corresponding 
holomorphic map by /an : -Z^an Z'^n- For an algebraic manifold Z and an Oz- 
module JF we set jF^n = Oz^,^ T . 

In the correspondence (1.1), let us consider also a functor in the derived category 
D^(C.) of sheaves of C- vector spaces, going in the opposite direction: 

r : D'^(Cy,J ^ D'^(Cx.J, r(F) = Rg,^J-^{F). 

For example, let L> be a Zariski locally closed subset of Fan and take F — Cd 
(the constant sheaf with fiber C on D and zero on Fan \ D): then, for any x & X 
one has 

riCo), ^ RrdSo,.; Cs,J, Sn,. ^ {y E D : (x,y) e S}. (1.4) 
One has the following "adjunction formulas" (see [5]). 

Proposition 1.7. Let C be a locally free Ox-'module and let F E D^(Cy-^^). Then, 
setting I = dim Y — dim S and m = dim S + dim Y — 2 dim X, we have 

Rr{X^;r{F)^C*J ~ RHom^^ {R{VCU, F ^ OyJ[1], (1.5) 

J an 

Rliom{r{F),C*J ~ RRom^^ JR{VCU, RHam{F,OYj)[m]. (1.6) 

Once the calculation of R{J)C) has been performed, these formulas will give 
different applications by computing r{F) for different choices of the sheaf F (a 
problem of geometric nature). 

1.4 Quasi-equivariant P- modules 

Let us recall the definition of (quasi- )equivariant "D-modules (we refer to Kashi- 
wara [7]). 

Let G be an algebraic group over C, and let g be its Lie algebra. We denote 
the enveloping algebra of q by U{^). Let Z he a G-manifold, i.e. an algebraic 
manifold endowed with an action of G. Let us denote by : G x Z — > Z the action 
IJ,{g, z) — gz and hy p : G x Z ^ Z the projection p{g, z) = z. Moreover, define 
the morphisms qj : G x G x Z ^ G x Z {j = 1,2,3) hj qi{gi,g2,z) = {gi,g-2z), 
q2{9i,92,z) = igig2,z) and q^igi, g2-, z) = {g2,z), and observe that fj, o = ^ o q2, 
poq2 =poq2 and A* o = p o g^. 

A G-equivariant O^-niodule is an O^-module M. endowed with an Ocxz-hnear 
isomorphism /3 : ijl*M. — > p*A4 such that the following diagram commutes: 

q2*l^*M — ^ q2*P*M 

\l , * 1^ 

qi*pL*M ^^qi*p*M ^ q3*pi*M^^ qz*p*M. 
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For a G-equivariant C^-module M. we have a canonical Lie algebra homomorphism 
PM-Q^ Endc(A^). 

Let Og^T^z denote the subalgebra OgxZ ®p-iOzP~^'^z of T^cxz- A P^-module 
Ai is called G-equivariant (resp. quasi-G-equivariant) if it is endowed with a G- 
equivariant O^-module structure such that the isomorphism /? : ii*A4 — > p*M. is 
"Dcxz-hnear (resp. Oq ^ P^-hnear). Note that for a morphism / : Z — > of 
algebraic manifolds and a P^z-module M. the P^-module H^{f~^M.) is naturally 
isomorphic to f*J^ as an C^-module. 

For example for a G-equivariant (9^-module the P^-module Vz ^ is 
endowed with a natural quasi-G-equivariant "D^-module structure. 

We denote by ModG(I^z) the category of quasi-G-equivariant D^-modules, and 
by 'Dq{T>z) the derived category of X>z-niodules with bounded quasi-G-equivariant 
cohomology (see Kashiwara-Schmid [8]). 

Let be a quasi-G-equivariant P^-module. The canonical Lie algebra homo- 
morphism — * Vz induces a Lie algebra homomorphism n^. '■ ~^ End£(A1). Set 

iM = PM — I^M ■ 

Proposition 1.8 (Kashiwara [7]). (i) We have 7x(a) G Endx>z{M.) for any 
a e 0. 

(ii) The linear map ■jm '■ d ~^ Endj)^{A4) is a Lie algebra homomorphism. 

(iii) We have jm — ^ if o-i^d only if Ai is G-equivariant. 
We also denote by 

-fM:U{Q)^Endvz{M) {1.7) 

the corresponding algebra homomorphism. 

Fix X & Z and set H = {g E G : gx — x}. For a G-equivariant C^-module A4, 
the fiber 

M{x) = C Mx 

of M. at X is endowed with a natural if-module structure. If is a quasi-G- 
equivariant P^- module, then A4{x) is also endowed with a g- module structure in- 
duced from the O^-hnear action For M — A4{x) we have the following. 

(a) the action of the Lie algebra of if on M given by differentiating the i?-module 
structure coincides with the restriction of the action of g, 

(b) hum = {Ad{h)u)hm for any h E H , u E q, m E M. 

Here Ad denotes the adjoint action. A vector space M equipped with structures of 
an iJ-modules and a g-module is called a (g, i7)-module if it satisfies the conditions 
(a) and (b) above. 

The following result plays a crucial role in the rest of this paper. 
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Proposition 1.9. Assume that Z — G/H, where H is a closed subgroup of G, and 
set X — eH e Z . 

(i) The category of G-equivariant Oz -modules is equivalent to the category of H- 
modules via the correspondence Ai ^ Ai{x). 

(ii) The category of quasi-G-equivariant Vz-modules is equivalent to the category 
of {q, H)-modules via the correspondence M. ^ M.{x). 

The statement (i) is well-known (see [11]), and (ii) is due to Kashiwara [7]. 

2 Radon transforms for generalized flag manifolds 

2.1 Queisi-equi variant "D- modules on generalized flag manifolds 

Let G be a connected reductive algebraic group over C, and Q the Lie algebra of G. 
The group G acts on q by the adjoint action Ad. Let [) be a Cartan subalgebra of 
0, A the root system in [)*, {ctj : i G /q} a set of simple roots, A"*" the set of positive 
roots, A~ the set of negative roots, = Hom(iJ, C^) C I)* the weight lattice, and 
W the Weyl group. For a e A we denote by the corresponding root space and 
by e f) the corresponding coroot. For i e Iq we denote hy Si E W the reflection 
corresponding to i. For w E W we set i{w) = ^(toA" fl A"^). Set p = | ^aeA+ ^i 
and define a (shifted) affine action of on f)* by 

w w{\ + p) - p. (2.1) 



For 7 C /o, we set 



A/ 


= AnE^«i> A+ = A/nA+, Wi = {si:iEl) 




iei 


li 




(f)z)/ 


= {A G f)^ : X{a^) ^ for any i G /}, 


(J)z)? 


= {Xet)*z-- A(an = for any z G /} C {i)*z)i, 


PI 


= ( E 




a6A+\A7 



We denote by wj the longest element of Wj. It is an element of Wj characterized 
by wi{Aj) — Af. Let L/, Nj and Pi be the subgroups of G corresponding to [/, U/ 
and pj. 

For A G (f)J)/ let V7(A) be the irreducible L/-module with highest weight A. We 
regard V7(A) as a P/-modulc with the trivial action of Nj, and define the generalized 
Verma module with highest weight A by 
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Let L(A) be the unique irreducible quotient of M/(A) (note that L(A) does not 
depend on the choice of / such that A e (f)^)/). Then any irreducible Pj-module is 

isomorphic to V/(A) for some A G and we have dimV/(A) = 1 if and only if 

A G (f)^)/- Moreover, any irreducible (g, P/)-module is isomorphic to L{X) for some 

A e mi- 
Let 

Xj = G/Pi 

be the generalized flag manifold associated to /. 

By the category equivalence given in Proposition 1.9 isomorphism classes of G- 
equivariant Oxj-modulcs (resp. quasi-G-equivariant Pxj- modules) are in one-to-one 
correspondence with isomorphism classes of P/-modules (resp. (5, P/)-modules). For 
A G (f)^)/ we denote by Cxj(A) the G-equivariant Oxj-module corresponding to the 
irreducible Pj-module Vi{\). We see easily the following. 

Lemma 2.1. LetX G (f)^)/. The quasi-G-equivariantVxj-module corresponding to 
the {q, Pi)-module M/(A) is isomorphic to 'DOxj(A) = Vxj <^q^ Oxji^)- 

We need the following relative version of the Borel-Weil-Bott theorem later (see 
Bott [3]). 

Proposition 2.2. Let I d J d Iq and let tt : X/ — > Xj he the canonical projection. 
For A G (1)^)/ ^6 have the following. 

(i) // there exists some a G Aj satisfying {\ + p — 2pi){a^) = 0, then we have 
P7r,((!?x,(A)) = 0. 

(ii) Assume that (A + p — 2p/)(a'^) 7^ for any a G Aj. Take w G Wj satisfying 
{w{X + p — 2pi)){a^) > for any a G Aj. Then we have 

P7r.(C»x,(A)) = OxAw{X + p- 2pi) - {p - 2pj))[-{i{wjw) - i{wi))]. 

Let /, J C /o with I ^ J. The diagonal action of G on x Xj has a finite 
number of orbits, and the only closed one G{ePj, ePj) is identified with X/nj = 
G/{Pi n Pj). In the rest of this paper we shall consider the correspondence (1.1) for 
X = Xi,Y = Xj and S = Xmj: 

Xj Xjnj — ^ Xj (2.3) 

and the Radon transform P('DOx/(A)) for A G Since / and g are morphisms 

of G- manifolds, the functor (1.2) induces a functor 

R:-Dl{Vx,)^-DUVxj). (2.4) 

Note that we have 

^9 - Oxjnjhi,j) for 7j,j = EaeA+\A, ^2-^) 
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2.2 Radon transforms of quasi-equivariant I>-modules 

Let A e We describe our method to analyze R{VOxj{X)) = g r^{VOxj{X)). 

By 

{r\VOxAmi<Pi n Pj)) ^ VOxAX)iePi) - M:{\) 

the quasi- G-equi variant Vxjf^j-Tcvodule f~^{VOxi{X)) corresponds to the (g, P/ fl 
Pj)-module M/(A) = U{q) (g^^-p ^ Vi{\) under the category equivalence given in 
Proposition 1.9. 
Set 

r = {x e Wi : X is the shortest element of Wic\jx}, (2.6) 
Tfc = {x e r : l{x) = k). (2.7) 

It is well-known that an element x G Wi belongs to T if and only if x~^^'\^j C A]!". 
This condition is also equivalent to 

{x{X + p)){a^) > for any a e A+^j. (2.8) 

In particular, we have x o X e {i)^)inj for x eT. 

By Lepowsky [9] and Rocha-Caridi [13] we have the following resolution of the 
finite dimensional Ij- module Vi{X): 

Q^Nn^ Nn-i ^ ,N,^No^ Vi{X) ^ (2.9) 

with n — dim Ij/li H pj and 

By the Poincare-Birkhoff-Witt theorem we have the isomorphism 

^i^i) '^{Ijnpj) ^inAx o A) ~ Uipi) «^(p^^^) Vinjix o A) 

of W([/)-modules, where ri/nj acts trivially on Vir]j{x o A). Moreover, the action 
of ri/ on U{pj) CS^(p^ ^) V/nj(a; o A) is trivial. Indeed, by [p/,n/] C xij we have 
njU{pj) = U{pj)nj, and hence 

ni{U{pi) «^(p^^^) Vinj{x o A)) C U{pi)ni (g) Vrnj(a; o A) C W(pj) (^n7Vrnj(2; o A) = 

by Tij C n/nj. Thus we obtain the following resolution of the finite dimensional 
pj- module Vi{X) (with trivial action of rij): 

^ at; ^ 7V;_^ ^ . . . ^ iV( ^ iV^ ^ Vj{X) ^ (2.10) 
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with 

By tensoring W(g) to (2.10) over U{pj) we obtain the following resolution of the 
(0, P/njj-module Mj(A): 

O^Nn^ Nn-i ^ , TVi ^ TVo ^ Mi{X) ^ (2.11) 

with 

Nk^^MinAxoX). 

Since the quasi-G-equivariant 'Dx/nj"^^'^^!^ corresponding to the (g, P/nj)-module 
Mir\j{x o A) is T>Oxjnj{x o A), wc have obtained the following resolution of the 
quasi-G-equivariant I'x/nj'^odule f^^{VOxji^))- 

O^Afn^ Nn-i ^ r\VOxA^)) ^ (2.12) 

with 

Mk=^VOx,,A^oX). (2.13) 

Our next task is to investigate on g^^Oxj^jix o A)) for x e F. We first remark 
that 

9Al^Ox,r.j{x o A)) = Vxj Rg.{Ox,r.j{x o A + 7/,j))- (2.14) 



Indeed, by (2.5) we have 



J 



^,(^C»x„,(x o A)) = Rg^Vx.^x,^, Px,n. Ox,^j{x o A)) 

= Rg, (Vxj^xrnj Ox,^, (x o A) ) 

= Rg^g-'Vxj ®g-.o^^ ^9 ®o^, , Ox.^^A^ o A)) 
= ^x, %*(^^p Ox.^A^ o A)) 

= ^x, ®o^^ Rg.{Ox,^A^ o A + 7j,j)). 

Lemma 2.3. Lei A G (t)J)/ and a; G F. 

(i) //(x(A+p))(q!^) = Q for some a G Aj, then we have Rg^{Oxjnj{xo^+'Ji,j)) — 
0. 

(ii) Assume that (x(A + p))(a^) 7^ /or an?/ a G Aj. Ta/se y G satisfying 
{yx{X + p))(q;^) > for any a G Aj . T/ien we /iave 

%,(e»x„.(^ o A + 7,,j)) = OxAiyx) ° >^)[-{^M - KmnA)]. 



Corrado Marastoni — Toshiyuki Tanisaki 



12 



Radon transforms for quasi-equivariant V-modules on generalized flag manifolds 



Proof. Since A+ \ Aj is stable under the action of Wj, we have ypj = pj for any 
y e Wj. In particular, 

pj = Sa{pj) ^ Pj- pj{oi^)a 

for any a G Aj, and hence pj{a^) — for any a e Aj. 
By the definition we have 

X o A + 77,j + p - 2pjnj = x{\ + p) + 77,j - 2pjnj = 2;(A + p) - 2pj, 

and 

y{x{\ + p) - 2pj) - (p - 2pj) = yx{X + p) - 2pj - (p - 2pj) = (yx) o A 
for any y G Wj. Hence the assertion follows from Proposition 2.2. □ 
Set 

r(A) = {x e r : {x{\ + p)){a') ^ for any a e Aj}, (2.15) 
Tk{\) = {x e r(A) : l{x) = A;}. (2.16) 

and for x e r(A) denote by y^ the element of Wj satisfying {yxx{\ + p)){a'^) > 
for any a e Aj. Set 

m{x) = i{wjyx) - i{wjnj) for x G r(A). (2.17) 

Lemma 2.4. For A e ^''^d ^ ^ ^(A) we have 

i{x) = e A+ \ A^ : {x{X + p))(a^) < 0}, (2.18) 
m{x) = e A+ \ Aj : {x{X + p))(q;^) > 0}. (2.19) 



Proof. We have 



e{x) = tt(x-^ AJ n A+) 

= tt{«eA+ : (a;(A + p))(a^)<0} 

= tt{ae A+\A^ : (x(A + p))(a^) < 0}, 



and 



m(x) = i{wj) - i{yx) - i{winj) 

= tl(A+\A,)-tt(y-iA7nA+) 

= tt(A+ \ A,) - G A+ : {x{X + p))(«^) < 0} 

= tl(A+ \ A,) - e An A/ : (:r(A + p))(a^) < 0} 

= tt{aeA+\A, : (a;(A + p))(a^) > 0} 

by (2.8). □ 
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Proposition 2.5. Let A G Then there exists a family {M.{k)*}k^o of objects 

o/Dq('Dxj) satisfying the following conditions. 

(i) A^(0)-~i?(I)Ox,(A)). 

(ii) M{ky ^0 fork> dim k/lj n pj. 

(iii) We have a distinguished triangle 

C{ky ^ M{ky ^ A^(A; + 1)' ^ 

C(fc)-= VOxA{yxx)oXMx)-m{x)]. 
Proof. For ^ A; ^ dim Ij/k n pj define an object Af{ky of D^(r>x,nJ by 

M{ky = [ >O^Afn^Afn-l^ >Afk^O---], 

where N'j has degree —j (see (2.12) and (2.13) for the notation). For k > dim [//l/fl 
pj we set Af{ky = 0. By 7\A(0)' ~ /-^(PCx,(A)) we have gM{oy ~ i?(PCx,(A)). 
Set Al(fc)* = gj^{ky. Then the statements (i) and (ii) are obvious. Let us show 
(iii). Applying g^ to the distinguished triangle 

Mk[k] M{ky M{k + 1)* ^ 
we obtain a distinguished triangle 

lMk[k] ^ M{ky ^ M{k+iy ^ . 
By (2.13), (2.14) and Lemma 2.3 we have 

£*-A4= vOxA{yxx)o\)[-m{x)]. 

The statement (iii) is proved. □ 
Theorem 2.6. Let A e (f)J)/. 

(i) We have 

p xer(A) 
m the Grothendieck group of the category of quasi-G-equivariantVxj -modules. 
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(ii) //r(A) = 0, then R{VOxj{X)) = 0. 

(iii) IfT{X) consists of a single element x, then 

R{VOxA^)) = VOxAiVxx) o X)[i{x) - m{x)]. 

(iv) If i{x) ^ m{x) for any x G r(A), then we have H^{R{VOxi{)^))) — unless 
p = 0. 

(v) If {X + p){a'^) < for any a e Aj \ A/, then there exists a canonical morphism 

$ : VOxAiwjWi^j) o A) ^ H\R{VOxAm- 

Moreover, $ is an epimorphism if i{x) > m{x) for any x e r(A) \ {e}. 

Proof. The statements (i), (ii), (iii) are obvious from Proposition 2.5. The state- 
ment (iv) follows from Proposition 2.5 and Lemma 1.4. Assume that A satisfies the 
assumption in (v). Then we have e e r(A) and ye — wjwmj- Hence (v) follows 
from Proposition 2.5. □ 

Lemma 2.7. (i) The map Wj x F — > WjWj ((y, x) i— > yx) is bijective. 

(ii) For A e {i)z)i we have 

{y^x : X e r(A)} = {w e WjWi : (w(A + p)){a^) > for any a e A+} 

and we have 

eix) - mix) = eiy,) + e{x) - tt(A+ \ A,) = i{y,x) - tt(A+ \ A,). 

Proof, (i) is a consequence of the definition of F, and the first statement in (ii) 
follows from (i) and the definition of y^.. By 

e{x) - m{x) = e{x) - {e{wj) - i{y,) - i{winj)) = i{x) + i{y,) - tt(A+ \ Ai) 

we have only to show i{yxx) — i{x) + i{yx) for x e F(A). We have 

xA^ n A" = xAj n A7 c A7 \ A/nj C A" \ Aj 

hj X e Wi and x~^Aj^j C A|. Since w e Wj, we obtain y^xA'^ n A~) C A~. 
Hence 

iiVxx) = ^iy.xA- n A+) 

= ^{y^xA- n A+) n A+) + Uy^xA- n A") n A+) 

= Ky.ixA- n A+) n A+) + ^{y^A- n A+) - tt(y,(xA+ n A") n A+) 

^e{x) + e{yx). 

□ 
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For A e (f)J)/ we set 

5(A) = {we WjWi : {w{X + > for any a e Aj^}. (2.20) 

Using Lemma 2.7 above we can reformulate Theorem 2.6 as follows. 
Theorem 2.8. Let A G (f)J)/. 
(i) We have 

in the Grothendieck group of the category of quasi-G-equivariantVxj -modules. 
(h) //5(A) = 0, then R{VOxj{X)) = 0. 
(hi) // 5(A) consists of a single element w, then 

RivOxA^)) = vOxA^ ° - ^i^j \ A/)]. 

(iv) Ife{w) ^ tt(A+ \ Ai) for any w e 5(A), then we have Hp{R{VOx,{X))) = 
unless p = 0. 

(v) //(A + p)(q;^) < for any a G Aj \A/, then there exists a canonical morphism 

* : VOxAi^J^inj) o A) ^ H\R{VOxAm- 

Moreover, $ is an epimorphism if £{w) > ti(Aj \ A/) for any w G 5(A) \ 
{wjwinj}. 

Remark 2.9. The following result which is a little weaker than Theorem 2.8(ii) can 
be obtained by observing that an integral transform for P-modulcs with equivariant 
kernel preserves the infinitesimal character of a quasi-equivariant P-module (see e.g. 
[8]): 

If (WoX)r] (f)J) J = 0, then R(VOxA^)) = 0. (2.21) 

An advantage of the argument using the infinitesimal character is that it also works 
for a broader class of integral transforms in equivariant contexts. 

Let us briefiy recall this argument (suggested to us by M. Kashiwara). Let Z be a 
G-manifold, denote by 3 (g) the center of U (g) and set n"*" = n0 = 0cte a+ Sa ^.nd n~ = 
©oeA- Sa- ^^y^ that a quasi-G-equivariant P^-module Ai has infinitesimal 

character % (for some x ^ Hom(3(0), C)) if 7A^(a) is the multiphcation by x(a) for 
any a G 3(0). Define a linear map a : 3(0) W(f)) ~ S{t)) as the composition 
of the embedding 3(0) — > U{q) and the projection U{g) — > W(f)) with respect to 



Corrado Marastoni — Toshiyuki Tanisaki 



16 



Radon transforms for quasi-equivariant V-modules on generalized flag manifolds 



the direct sum decomposition — W(f)) © (n~W(0) +W(0)n"'"). Then a is an 

injective homomorphism of C-algebras. For A e f)* define an algebra homomorphism 
X\ '■ 3(0) — ^ C by XA(a) = ((T(a),A). By a result of Harish-Chandra, any algebra 
homomorphism from 3(g) to C coincides with x\ fo^^ some A G [)*, and for A,/i G P)* 
one has x\ = Xn if ^-iid only if /i E W o X. By the category equivalence of Proposition 
1.9, the infinitesimal characters of quasi- G-equi variant "Dxj-niodules are of the form 
X\ for A G Therefore, recalling Harish-Chandra's result, if {WoX)r\{i)^)j — 0, 

then R{VOxj{X)) = 0. 

2.3 Extremal cases 

We characterize the extremal cases (see Definition 1.5) in the correspondence (2.3). 
We shall only deal with the invertible (9-modules. Given A G (f)z)/ and /j, G (f)z)j, 
we write for short (A,yu) instead of {Oxi{X),Oxj{fJ'))- 

Proposition 2.10. The pair (A, fi) is an extremal case if and only if /j, — X + ^i^j. 

This condition is also equivalent to the following system 



Proof. The first statement is obvious by (2.5). Since \ A/ and Aj are stable 
under the action of Wi and Wj respectively, we have wl'ji^j) = ji^j for any w G 
Winj — Wj n Wj. In particular, we have 

= Sii'j) = 77,j - 7/,j(a,^)ai 

for any i G / fl J. Hence we obtain 



By (2.22) we have the following 

Corollciry 2.11. If q is semisimple and I \J J — Iq, there exists a unique extremal 
case for (2.3). 

Proposition 2.12. If {X, n) is an extremal case, then we have 



and (wjwinj) o X — /i. In particular, we have e G r(A) and i{e) — m(e) = 0. 




(2.22) 



7/,j(«i^) = for any i G / fl J. 
Therefore, the relation /i — X + is equivalent to the system (2.22). 



□ 




for any a G Aj \ A7, 
for any a G A'^, 
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Proof. Since // and 77,7 are fixed by the action of Wj and Wmj respectively, We 
have 

{wjwinj) o A = wjwif^j{ii - 'yi^j + p) - p = ^ - wj{'yi^j - wmjp + wjp). 

By 

winjp - wjp = (p - wjp) - (p - winjp) ^ ^ 01 - ^ ol^Ii.j 

we obtain {wjwmj) o X — p,. Hence by wjwinj{^j \ A/) C Aj and p e (^z) j, we 
have 

(A + = {winjWj{p + = {p + p){wjWinja^) < 

for any a G Aj \ A/. Moreover, we have (A + > for any A/ by (2.22). □ 

By Proposition 1.6, if the pair {X, p) is an extremal case we get a nontrivial 
Pxj-linear morphism 

$ : VOxAfJ') ^ H%R{VOxAm- (2-23) 
Theorem 2.13. Let {X, p) be an extremal case. 

(i) We have Hp{R{VOxi{^))) — Z*^*^ ^''^U P ^ '^f only if i{x) ^ m{x) for 
any x G r(A) . 

(ii) Assume that H^{R{VOxi{^))) — /^'^ any p 7^ 0. Then $ is an epimorphism 
if and only if i{x) > m{x) for any a; G r(A) \ {e}. 

(iii) Assume that Hp{R{'DOxj{^))) — for any p ^ 0. Then $ is an isomorphism 
if and only ifr{X) = {e}. 

We need the following result in order to prove Theorem 2.13. 

Lemma 2.14. Let {X, p) be an extremal case, and let a;i,a;2 G r(A). Set = yx^ 
for k — 1,2. If L{{y iXi) o A) appears as a subquotient of Mj{{y2X2) o X), then we 
have e{x2)-e{y2)Si{xi)-e{yi). 

Proof. For ^ G f)^ we set 

A+(0 = {ae A+ : (e + p)(a^)=0}, Wo{0 = {w e W -.100^ = ^}. 

Take v eWoX such that {1/ + ^ for any a G A+, and let w e W such that 

X = w o u. We can assume that i{w) ^ i{x) for any x & W satisfying X — x o u. 
Then w is the (unique) element of wWoIp) with minimal length. 
Let us first show: 

ykXkW is the element of ykXkwWQ{i') with minimal length. (2.24) 
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It is sufficient to show ykXkW^Q^u) C A+. Since w is the element of wWq^u) 
with minimal length, we have wAq (z/) C A"*", and hence wAq (z/) = Aq (A). By 
Proposition 2.12 we have A^(A) C A+ \ A/. Hence by Vr/(A+ \ A/) = A+ \ A/ 
we have ,TfcA([(A) C A'^. Thus a;^.A(]"(A) = A([(a;fe o A). By xj. G r(A) we have 
Aq (xfc o A) C A+ \ Aj, and hence yk/^^{xk o A) C A+. The statement (2.24) is 
proved. 

We next show 

^{.ykXkw) = ^{w) + l{xk) - iiVk). (2.25) 

For any a e Af we have 

(i/ + p)(«;-V^) = (A + p)(a^)>0, 

and hence w~^A^ C A+ by the choice of u. Thus wc have 

w'^ix^^A-^ n A") = w"^(a;^^A| n A7) c w'^Aj c A". 

Hence £{xkw) — i{w) + £{xk). Here, we have used the well-known fact that for 
u,v E W we have £{uv) — i{u) + i{v) if and only if u{vA'^ fl A~) C A~. Similarly, 
we have 

(z/ + p)(ti;"^a;^^i/^^a^) = (^^^^(A + p))(a^) > 

for any a G Aj by the definition of yk and hence w~^x'^^y^^Aj C A"*". Thus we 
have 

^''^kViVk^'^ n A-) = w-'x^'y^\ykA+ D Aj) c w-^x^^t/.-^Ay c A". 

Hence i{xkw) — i{ykXkw) +£{yk). The statement (2.25) is proved. 

Note that L{{yiXi) o A) = L{{yiXiw) o u) and that Mj{{y2X2) o A) is a quotient 
of the ordinary Verma module M{{y2X2'w) 01/) = M(D{{y2X2w) o u). Hence by our 
assumption and by (2.24) we obtain yiXiW ^ y2X2W with respect to the standard 
partial order on Vl^ by a result of Bernstein-Gelfand-Gelfand [2] concerning the 
composition factors of Verma modules. In particular, we have i{yiXiw) ^ i{y2X2w). 
Hence we obtain the desired result by (2.25). □ 

Proof of Theorem 2.13. We shall use the notation in Proposition 2.5. 
We first show the following. 

If H'-iMiky) = for any k^£, then H'{C{ky) = for any k^E. (2.26) 

Assume that there exists some k ^ i such that H^[C{k)') 7^ 0. Let ko be the largest 
such k. Then wc have exact sequences 

H'-\Miko + 1)-) ^ H'iCiko)') ^ 0, (2.27) 
H'-\C{ky) ^ H'-\M{ky) ^ H'-\M{k + 1)*) ^0 {k> ko). (2.28) 
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By H^{C{ko)*) 7^ there exists some xi e r(A) such that £{xi) — m{xi) = — r, 
i{xi) — ko and VOxjiiVxiXi) o A) is a direct summand of H^{C{ko)*). On the 
other hand by (2.27) and (2.28) any irreducible subquotient of H^{C{k(])*) is isomor- 
phic to an irreducible subquotient of H^^^{C{k)*) for some k ^ k^ + 1. Moreover, 
H^~^{C{k)') is isomorphic to the direct sum ofVOxj{{yx2^2)° ^) for 2:2 G r(A) such 
that £(^2)— m(x2) = —{r—l),i{x2) = k. By the category equivalence given in Propo- 
sition 1.9 we see that there exists some X2 G r(A) such that i{x2) —m{x2) — — (r — 1), 
i{x2) ^ /co + 1, and that L{{yxj^Xi) o A) is isomorphic to an irreducible subquotient 
of Mj{{yx2X2) o A). Then by Lemma 2.14 we have 

£{x2)-£{y.,)^£{xi)-l{yx,). (2.29) 

On the other hand we have 

£{X2) + £{y.,) = £{xi) + l{yx,) + 1. (2.30) 

by Lemma 2.7. Hence we have 2l{x2) ^ 2£(a;i) + 1. Since l{xi) and (-{X2) are 
integers, we obtain l{x2) S -^{xi). This is a contradiction. The statement (2.26) is 
proved. 

Let us show (i). By Theorem 2.6(iv) we have Hp{R{VOxj{\))) = for any 
p^Oii i{x) ^ m{x) for any x G r(A). Assume that HP{R{VOxji\))) = for any 
p > and that £{x) < m{x) for some x e r(A). Then we have if^(A1(0)*) = for 
any p > and HP{C{k)') ^ for some p > and some A; ^ 0. Let r be the largest 
positive integer such that H'^{C{k)*) ^ for some /c ^ 0. Then we have an exact 
sequence 

H'{M{ky) ^ H'{M{k + 1)') ^0 (k^O). 

Since H'^{M{0)') = 0, we see by induction on k that H^{M.{k)*) = for any k ^0. 
Hence by (2.26) we have H^{C{k)*) = for any A; ^ 0. This is a contradiction. The 
statement (i) is proved. 

Let us show (ii). By (i) and the assumption we have i{x) ^ m{x) for any 
X e r(A); in other words HP{C{ky) = for any p > and any k ^ 0. By 
Theorem 2.6(v) $ is an epimorphism if i{x) > m{x) for any x G r(A) \ {e}. Assume 
that $ is an epimorphism. Since $ : H^iCif))*) H^\A4{0)*) is an epimorphism, 
we have H^{M.{ky) = for any A; > by the exact sequences 

//°(C(0)') ^ //°(A^(0)*) ^ //°(M(1)') ^ 0, 
H\M{ky) ^ H\M{k + !)•) ^ 

Hence by (2.26) we have H^{C{ky) = for any A; > 0. It implies that i{x) > m{x) 
for any x G r(A) \ {e}. The statement (ii) is proved. 

Let us finally show (iii). By (i) and the assumption we have HP{C{ky) = for 
any p > and any A; ^ 0. By Theorem 2.6(v) $ is an isomorphism if r(A) = {e}. 
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Hence it is sufficient to show that H~P{C{k)') — for any /c > and any p ^ 
if $ is an isomorphism. Let us show it by induction on p. If p = 0, then we have 
H^{C{ky) = for any A; > by the proof of (ii). Assume that the statement is 
proved up to p. Consider the exact sequence 

H-^+^\M{oy) H-^+^\M{iy) H-p{c{oy) H~p{M{oy). 

We have H'P{C{Oy) = for p > 0, and $ : iJ-P(C(0)*) ^ H-P{M{Oy) is an 
isomorphism for p = 0. Moreover, we have H^^P^^\Ai(Oy) = by Lemma 1.4. 
Hence we have H-^+'^\M{iy) = 0. Thus we obtain H-^+^\M{ky) = for any 
A; > by the exact sequence 

H-^+'\Miky) H-^P+^\Mik + 1)') ^ H-P{C{ky) 

and the hypothesis of induction. Hence we have H~^~^^\C{ky) — for any A; > 
by (2.26). The statement (iii) is proved. □ 
By using Theorem 2.13 (i) and a case-by-case analysis we obtain the foUowing. 

Theorem 2.15. Assume that G is a simple group of classical type and that {!(/) = 
tl( J) = tl(/o) — 1. Let (A, /i) be the extremal case. Then we have Hp{R{VOxj{)^))) — 
unless p = 0. 

Details of the proof is omitted. We do not know an example of an extremal case 
(A,/x) satisfying Hp{R{VOxj{X))) 7^ for some p > 0. 

Remcirk 2.16. Let (A, n) be an extremal case. For x eT and a e Aj \ A/ we have 

{x{X + p)){a'^) = (A + xp)){a^) = {p - 7/,J + xp){a'^) = {xp - 77,j)(a'^), 

and hence we have Hp{R{VOxi{X))) — for any p > if and only if 

r for X e r satisfying {xp - 7/,j)(q;^) ^ for any a G A \ A/ (o '^^\ 

\ we have ^{a e A+ \ Aj : {xp - li,j){a') > 0} ^ ^{x) . ^ ' 

In the next section we shall give conditions in order that $ is an cpimorphism and 
that $ is an isomorphism under the assumption of Theorem 2.15. In particular, $ is 
not necessarily an epimorphism nor a monomorphism. It seems to be an interesting 
problem to determine the kernel and the cokernel of 

Remark 2.17. In Tanisaki [15] it is shown in certain cases that Ker$ corresponds 
to the unique maximal proper submodule of Mj{p) under the category equivalence 
given in Proposition 1.9. 
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3 The maximal parabolic case for classical simple groups 

In this section we apply our results to the case where G is a classical simple group 
and Pi, Pj are maximal parabolic subgroups, and obtain results for the Radon 
transform R{VOxi{^)) '^i^^ respect to the geometric correspondence 

-\7' f \r 9 \r 

-^i ■* -^inJ -^J 

for A e (f)z)?- this case we have 

I — Io \ {p} and J — Io \ {q} for some p ^ q, (3.1) 

and (f)^)? — {t^'^p '■ £ '^}, where zuk denotes the fundamental weight corresponding 

to A; G Iq. 

We keep the standard notations of Bourbaki [4]. In particular, if G is of rank n, 
then /o = {1, 2, . . . , n}. 

3.1 The case (A„) 

In this subsection we consider the case where G — SL{V) for an n + 1-dimensional 
complex vector space V. By the symmetry of the Dynkin diagram we may (and 
shall) assume that p > q. We have the identifications: 

Xi — {p-dimensional subspace oiV}, 

Xj — {g-dimensional subspace olV}, 

Xinj = {(C/i, U2) eXjxXj-.UiD U2}, 

and /, g are natural projections. The invertible Oxj-modulc Oxji^p) corresponds 
to the tautological line bundle whose fiber at G Xj is /\^^ U (a subbundle of the 
product bundle Xi x /\^V), and we have Oxi{TZUp) = Oxji'^p)'^^ ■ Hence in the 
standard notation of algebraic geometry we have Oxi{rzUp) = Oxii^r). 
For k E lo — {l, - ■ ■ iTi} set 

k^ — n+l — k, k+ — max{k,k^}, k- — mm{k, k^,}. 

We first give consequences of Theorem 2.6. A weight A — Yl^=i ^i^i i^i ^ 
= 0) belongs to (f)J) j if and only if Ai ^ • • • ^ Ag and A^+i ^ • • • ^ A„+i. 
The Weyl group W is identified with the symmetric group Sn+i, and it acts on the 
weights by permutations of the components, i. e. crA = Y17=i ^i^<T{i) ^ ^ ^ ■ 

Then we have Wi = Sp x Sp^ and Wj = Sg x Sg^. We have 

= [(n + 1 - p) (£1 H Ep) - p{ep+i H Sn+i)] 

= £1 H h £p + const(£i H h £n+l) 

P = ^ [nei + {n- 2)^2 H h {-n)en+i] 

= -£2 nSn+i + const(£i H h £n+l), 
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and therefore we get 

rvjp + p = r£i + (-1 + r)e2 H h (-(p - 1) + r)ep - pSp+i nSn+i 

+ const (£i H h £n+l)- 

By the assumption q < p the set r(rrop) consists of (cr, r) e Sp x Sp^ satisfying 

T = 6 

a-\l)<---<a-\q), 

{a-\q + 1), . . . , (7-i(p)} n {p + r + 1, . . . , n + r + 1} = 0, 

and we have 

£(((7, e)) =tt{(a, b) ■.l^a^q,q + l^h^p,a-\a)> a-\b)}, 
m{{a, e)) c) : gr + l^fe^p, p + + (7~^(6) <r + c}. 

Hence by Theorem 2.6 we obtain the following results. 

Proposition 3.1. (i) Assume q < p^. Then we have R{'DOxi{—a'UJp)) = if 
q + l^a^q^-1. 

(ii) Assume q^p^. Then we have 

R(VOxA-<l*^p)) =VOxj{-p.Wg), 
R{VOxA-<l^v)) =VOxA-pzu,)[-{p - q){p, - q)]. 

(iii) H''{R{VOxj{-avjp))) ^ for any k if a ^ q^. 

Let us consider the extremal case. By 

p n+l 
i=q+l i=p+l 

and (2.22) the extremal case is given by {—q^Wp,—p^Wq). By Theorem 2.13 we 
obtain the following. 

Proposition 3.2. We have H^{{R{VO Xi{—q*'OJp))) — for any k 0, and there 
exists a canonical nontrivial epimorphism 

$ : VOxA-P*^,) ^ H\R{VOxA-Q*^p)))- 

Moreover, $ is an isomorphism if and only if p* ^ q. 
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Remcirk 3.3. In the situation of Proposition 3.2 it is proved in [15] that for p* S Q 
the kernel of $ is the maximal proper G-stable submodule of T>Oxj{—p*'^q)- 

In the rest of this subsection we assume that q < p_ and give application to 
topological problems. By Proposition 3.1 we have 

{VOxj{-p*T:Ug) for a = g*, 

for g + 1 < a < - 1, (3.2) 

VOxj(-prVq) [-Ipq] for q, 

where Ipg = (p — q){p* — q)- Thus by Proposition 1.7 we have the following. 

Proposition 3.4. For any F e 'D^{£.Xj^^) and g+l^a^g* — 1 we have 

,an) ^ 

RRom{r{F),OxAazUp),n) = 0, 

and for {a,b,c,d) = {q*,p*, (p - q)p*,PP* - qq* - q{p - q)) or {a,b,c,d) = {q,p,q{p- 
~Qip ~ q)) have 

Rr{Xj,,^;r{F)^OxA ) ~ Rr{Xj,,^;F ^OxAbWgU)[-c], 

RHom(r(F),C»x,(ati7p)an) ^ RHom (F, C>x^(6ca,)a„)[-d]. 

Let us treat some particular cases. In the following we set N = qq^.. 

(1) Let yo e Xj, and set F — '^{y^}- Since g~^{yo) ^i,yo ^ closed embedding, 
one has 

r(F)^Cx,„„„„, (3.3) 

where Xj^y^ = fg~^{{yo}) = {x E Xj : yo <Z x} (identified with the Grassmannian 
of {p — g)-subspaces of V/yo). By Proposition 3.4 and (3.3) we obtain the following. 

Proposition 3.5. For any g+l^a^g* — 1 we have 

Rr(x,,,„,an; OxAo'^vU) ^ 0, Rrx,„„,.„(X/; OxAa-^vU) ^ o, 

and for (a, c, d) = {q^, {p - q)p*,pp* - qq* + p*q) or (a, c, d) = {q, q{p - q),p*q) we 
have 

H%Xj,y^,,^; OxAa^pU) ^ C{z}, H^x,,,^ jXi; OxAa^pU) ^ B^^. 

where C{z} (resp. B^^^) is the ring of convergent power series in z — {zi, . . . , zn) £ 
(resp. the ring of hyperfunctions in along {0} of infinite order), and all other 
cohomology groups vanish. 
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Namely, one identifies RT{Xj^^;Cy„ ®Oxj{hwq)^^) ~ Rr({0}; Cc^v ) = C{z} and 
RHom(C,„; OxA^^.U) ^ Rr{o}(Cf„; Oc|. ) = B^^^-N]. 

(2) Let Zo be a g*-subspace of V, = {y & Xj : |/ fl = 0} ~ and set 
^ = <Ci5,o,an- One has 

riF)^CE.^J-Hp-Q)], (3.4) 

where E^^ = fg~^{E;,^) — {x G Xj : dim(a; fl 2:0) = p — q} (i-c. the p-dimcnsional 
subspaces of V in generic position w.r.t. Zo). Namely, the map / = ))an • 

{g~^{Ez^))g,n -£"20, an is a complex vector bundle of rank q{p — q) (the fiber over 
X E Eza is Se^^,x = {y E Ez„ : y c a;} ~ O^*'"'?)); hence there is a morphism 
of functors Rf ^f^^\2,q{p — q)] — > idDb(-c__ ) defining a natural morphism r{F) = 

^zo ,an 

Rfa.n\^{g-HEzo)).n '^B,„.an[~2g(p - g)], which is an isomorphism since, by (1.4), 

one has r{F)x — C[— 2g(p — q)] (for x G £'zo,an) and = (otherwise). 
By Proposition 3.4 and (3.4) we obtain the following. 

Proposition 3.6. For any g+l^a^g* — 1 we have 

and for (a, c, d) = {q*,p{p* - q) + q'^,P*ip - q)) or (a, c, d) = (g,p*g, qip - q)) we 
have 

^^:(^.„,a„;0X,(aU7,),,) ^ i/e'^(£^.„,an;C»£;.„,.J, 

i^'(^.o,an;C»x,(ati7,),„) ~ r(£;,„,,„;C»s^^,,j 

y;/iere i/f (i?,„,,„; 0^,^,,J ^ r(i?,„,,„; fi^,^,,J' (resp. r(i?,„,,,; are Mar- 

tineau's analytic functionals (resp. the entire functions) in £^zo,an — , o-nd all 
other cohomology groups vanish. 

Namely, one has RT{Xj^^,,- C^,, .„®C»x,(6c^,)an) ^ H^{Ez^,.n; Oe,^ J[-N] and 
RHom(CE.,,.„;Ox,(6t^,)an) ^ r(^,,,a,;C»E.„,.J. 

3.2 The case (B„) 

In this subsection we consider the case where G is (the universal covering group 
of) SO{V) for an 2n + 1-dimensional complex vector space V equipped with a 
non-degenerate symmetric bilinear form ( , ) : y x y — > C. Then we have the 
identifications: 

Xi — {p-dimensional subspace U olV such that (C/, U) — 0}, 
Xj — {g-dimensional subspace U olV such that (C/, U) — 0}, 

^ r {{UiM2)eXixXj:U,(ZU2] {p<q) 
'''' \ {(C/i, U2) eXixXj-.UiD U2} {p>q), 
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and /, g are natural projections. The invertible Oxj-module Oxji'cup) corresponds 
to the tautological hne bundle whose fiber at C/ E Xj is /\^ U. 
By Theorem 2.6 we have the following. 

Proposition 3.7. (i) We have R{T>Oxi{—a'^p)) — in the following cases: 

2n — p — q<a<q if p < q^n, 

min(n — p,q)<a<n — q if q < p < n, 

n — p + q<a< max(n, 2n — p — q) if q < p < n, 

2q < a < 2{n — q) if p = n, 

(ii) We have R{'DOxj{—azUp)) — VOxji—bzUgjl—c] in the following cases: 

(a, b, c) 

0) {p < q < n, 2n — 2p — q ^ 0), 

{2n-p- q, 2{n - q), Ci) {p < q ^ n, 2n - 2p - q ^ 0), 

(n,2p,0) {q^n,n-2p^0), 

(2n — p — q,2n — p — q,0) {q < p < n, 2n — 2p — q ^ 0) , 

, {q,P,C2) {q < p < n, 2n -2p - q^O), 

where 

(q — p) (3p + 3g — 4n — 1) ip — q) [in + 1 — 3p — 3g) 

2 ' 2 ■ 

By Theorem 2.13 we have the following. 

Proposition 3.8. Let 



(r, s) = 



{q.p) ifl^p<q^n — 1, 

(2n — p — q,2n — p — q) ifl^q<p^n — 1, 

(2(n — q),n — q) ifp = n, l^q^n — 1, 

{n, 2p) if 1 ^ p ^ n — 1, q = n. 



Then we have H''{R{VOxj{—rzUp))) = for any k ^ 0, and there exists a canonical 
nontrivial morphism 

$ : VOxA-sw^) ^ H\R{VOxA-rr:Op)))- 
Moreover, $ is an epimorphism if and only if we have either 

(a) p<q^n, 

(b) q < p < n and 2n — 2p — q ^ 0, 

and an isomorphism if and only if we have either 

(a) p < q ^ n and 2n — 2p — q ^ 0, 

(b) q < p < n and 2n — 2p — q^ 0. 
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3.3 The case (Cn) 

In this subsection we consider the case where G = Sp{V) for an 2n- dimensional 
complex vector space V equipped with a non- degenerate anti-symmetric bilinear 
form ( , ) : V X V ^ C Then we have the identifications: 

Xj = {p-dimensional subspace U of V such that (f/, U) = 0}, 
Xj = {g-dimensional subspace U of V such that {U, U) = 0}, 

{(f/i, U2) eXixXj:U,G U2} {p < q) 

{(f/i, f/2) e X, X : [/i D U2} {p>q), 



and /, g are natural projections. The invertiblc Oxj-module Oxji'^p) corresponds 
to the tautological line bundle whose fiber aX U & Xj is /\^ U . 
By Theorem 2.6 we have the following. 

Proposition 3.9. (i) We have R{VOxj{—(i'!^p)) = in the following cases: 

2n — p — q+l<a<q if P < 

q<a<2n — p — q + 1 if q < p- 

(ii) We have R{VOx,{—ci'^p)) — Xj{—bva ^[—c\ in the following cases: 
(a, 6, c) 

(g, p, 0) (p < g ^ n, 2n — 2p — g + 1 ^ 0) , 

(2n — p — g + 1, 2n — p — g + 1, ci) (p < g ^ n, 2n — 2p — g + 1 ^ 0), 
(2n — p — g + 1, 2n — p — g + 1, 0) (g < p ^ n, 2n — 2p — g + 1 ^ 0), 
(g,p,C2) (g<p^n, 2n-2p-g+1^0), 

where 

(g — p) (3p + 3g — 4n — 1) (p — g) (4n + 1 — 3p — 3g) 

C] = , Co = . 

2 ' 2 

By Theorem 2.13 we have the following. 
Proposition 3.10. Let 



{q,p) if'i-^P<q^n, 
{2n — p — q + l,2n — p — q + 1) if 1 ^ q < p ^ n. 



Then we have H^^RiVOxi^—TVOp))) = for any k ^ 0, and there exists a canonical 
nontrivial morphism 

$ : VOxA-s^,) ^ H\R{VOxA-rzUp))). 
Moreover, $ is an epimorphism if and only if we have either 
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(a) p < q < n and n — p — q ^ 0, 

(b) p < q ^ n and 2n — 2p — q + 1 ^0, 

(c) q<p^n, 

and an isomorphism if and only if we have either 

(a) p < q ^ n and 2n — 2p — q + 1 ^0, 

(b) q< p^n and 2n - 2p - q + 1 ^0. 

Remark 3.11. In the situation of Proposition 3.10 it is proved in [15] that Ker$ is 
the maximal proper G-stable submodule of VOxji—szUg) ii q — n and 2p ^ n — 1. 

3.4 The case (D„) 

In this subsection we consider the case where G is (the universal covering group 
of) SO{V) for an 2n-dimensional complex vector space V equipped with a non- 
degenerate symmetric bilinear form { , ) : V x V ^ C 
For 1 ^ A; ^ n set 

X{k) — {/c-dimensional subspace U oiV such that (C/, U) — 0}. 

Then X{k) is connected for 1 ^ /c ^ n — 1, and X{n) has two connected components, 
say Xi{n) and X2{n). Then we have the identification: 

X(A;) = X,„\|fc} {l^k^n-2), 

X{n-1) = Xi^\^n-l,n}, 

Xi{n) = 

X2{n) = Xjg\{n-1}- 

If {p, q} ^ {n — 1, n}, then 

^ r {{Ui,U2)eXjXXj:UiCU2} {p<q) 
^^■^ \{{U^, U2) eXjxXj:U^^ U2} {p>q), 

and if p = n — 1 and q = n, then / (resp. g) assigns U G Xj^-^j = X(n — 1) to 
the unique U' e Xj = X2{n) (resp. U' e Xj = Xi{n)) such that U C U'. The 
invertible Oxj-module Oxii'^p) corresponds to the tautological line bundle whose 
fiber at U e Xj is /\ U where A;=pforl^A;^n — 2 and A; = n for p e {n — 1, n}. 
By Theorem 2.6 we have the following. 
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Proposition 3.12. (i) We have R{T>Oxi{—aWp)) = in the following cases: 



2n — p — q — 1 < a < q 
q<a<2n — p — q — 1 
2q < a <2{n - q -1) 
n — p — l<a<n 
a — n — 1 



if p < q ^ n — 2, 
ifq<p^n — 2 

if p e {n — 1, n}, 1 ^ q ^ n — 2, 
^/ 1 = P = ^ ~ 2, g e {n — 1, n}, 
if {p, q} — {n — l,n} and n is even. 



(ii) We have R{T>Oxj{—aWp)) = VOxj{—bzUg)[—c] in the following cases: 



(a, b, c) 

{q,p, 0) 

(2n — p — q — 1, 2n — p — q — 1, ci) 
{n,2p,0) 

{n-p-l,2{n-p - 1), C2) 
< (2n — p — q — l,2n — p — q — 1,0) 

(ra,n-2,0) 
(n — 1, n — 1, 0) 
(n - 2, n, 0) 



where 



Cl 



C3 



(g — p) (3p + 3g — 4n + 1) 

2 ' 
(p-g)(4n-3p-3g-l) 



By Theorem 2.13 we have the following. 
Proposition 3.13. Let 

(r, s) 

{q,p) 

(2n — p — g — l,2n — p — g — 1) 
(2(n — g — l),n — g — 1) 
(n,2p) 
I (n,n-2) 



(p < q ^ n - 2, 2n - 2p - q - 1 ^ 0), 

Ip < q S n - 2, 2n - 2p - q ~ 1 S 0), 

{p ^ n - 2, q e {n - l,n}, n - 2p - 1 S 0), 

{p ■^n-2, q e {n - l,n}, n-2p - 1 ^0), 

[q < p S n - 2, 2n - 2p - q - 1 ^ 0), 

{q < p ^ n - 2, 2n - 2p - g - 1 ^ 0), 

(.{P,q} = {n- l,n}, n : odd), 

(feg} = {n-l,n}, n : odd), 

{{P,(l} = {n- l,n}, n : odd). 



C2 



{n — p){3p — n + 1) 



ifl^q<p^n — 2, 
if p E {n — 1, n}, 1 ^ q ^ n — 2, 
^/ 1 = P = ^ ~ 2, g e {n — 1, n}, 
if{p,q} = {n-l,n}. 



Then we have H^{R{VOxj{—rzUp))) = for any k 0, and there exists a canonical 
nontrivial epimorphism 

$ : VOxj{-sw,) ^ H%R{VOxA-r^p)))- 
Moreover, $ is an isomorphism if and only if we have either 



Corrado Marastoni — Toshiyuki Tanisaki 



29 



Radon transforms for quasi-equivariant V-modules on generalized flag manifolds 

(a) p < q < n — 1 and 2n — 2p — q — 1 ^ 0, 

(b) q < p < n — 1 and 2n — 2p — q — 1 ^ 0, 

(c) p < n — 1 , q & {n — 1, n} and n — 2p — 1 ^ 0, 

(c) {p, q} = {n — l,n} and n is odd. 

Remark 3.14. In the situation of Proposition 3.13 it is proved in [15] that Ker<l> is 
the maximal proper G-stable submodule ofVOxj{—sWq) if g e {^ — 1, n}, 2p ^ n—2 
and if g = l,p G {n — 1, n}. 
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